Conversation
The a16w16 split-K opus kernels keep their partial-sum buffer in a per-stream hipMalloc registry on gfx942/gfx950. Any HIP graph capture that is the first thing to touch it aborts: [AITER] opus_gemm.cu:487 splitk workspace not initialized for the current CUDA stream. Call aiter.opus_gemm_workspace_init() ... Frameworks that capture cudagraphs (vLLM, SGLang) never make that call, so every tuned row selecting libtype=opus on these two archs is a latent engine crash for them. gfx1250 is unaffected -- ROCm#4246 gave it a torch.empty workspace that is capture-aware -- so its 127 opus rows are kept. Removes 259 rows (183 gfx942 + 76 gfx950); those shapes fall through get_GEMM_A16W16_config to asm/skinny/torch, all capture-safe. Deletions only, no row is edited. Reproduced on Kimi-K3 MXFP4 TP8 / 8x MI355X (gfx950) under vLLM: capture dies at 58% of the PIECEWISE ladder on M:20/19/18 N:6288, and after warming the workspace on the capture stream it merely relocates to FULL capture -- the per-stream registry cannot be reliably pre-registered from the framework side. Note the failure is concurrency-dependent: at concurrency 1 the ladder never reaches an opus shape, so low-concurrency smoke tests pass. This is a mitigation, not the root fix. Porting ROCm#4246's torch.empty workspace to the gfx942/gfx950 split-K kids would let these rows come back. Signed-off-by: hyukjlee <hyukjlee@amd.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
PR title tags: |
|
We'll fix this at the root — porting the workspace to torch.empty (capture-safe) for gfx942/gfx950 split-K kids, same as the prewarm path already does for the capture stream. That avoids the hipMalloc-inside-capture abort without sacrificing any tuned performance. This PR deletes 259 opus rows, but only 70 of them are split-K (the ones that actually touch the per-stream hipMalloc registry). The other 189 rows are non-splitk kids (flatmm/persistent/mono_tile) that never touch the workspace and are capture-safe — removing them drops tuned performance for no reason. |
Motivation
The a16w16 split-K opus kernels keep their partial-sum buffer in a per-stream
hipMallocregistry on gfx942/gfx950. Any HIP graph capture that is the first thing to touch that registry
aborts:
[AITER] opus_gemm.cu:487 splitk workspace not initialized for the current CUDA stream.
Call aiter.opus_gemm_workspace_init() inside with torch.cuda.stream(s): ...
Frameworks that capture cudagraphs (vLLM, SGLang) never make that call — vLLM contains no
opus_gemmreferences at all — so every tuned row selectinglibtype=opuson these two archsis a latent engine crash for them. The row itself is fine in eager mode; it is only unusable
under capture.
gfx1250 is unaffected: #4246 gave it a
torch.emptyworkspace (_get_opus_workspace) that isHIP-graph-capture aware, and its docstring states "No eager pre-warm is required." Its opus
rows are therefore kept.
Technical Details
Removes 259 rows where
libtype == "opus"andgfx ∈ {gfx942, gfx950}:torch.empty, capture-safe (#4246)every surviving byte is untouched.
get_GEMM_A16W16_configto its default branch (asm / skinny /torch), all capture-safe.
("gfx", "cu_num", ...), removing a row only affects thearch it names.
This is a mitigation, not the root fix. Porting #4246's
torch.emptyworkspace to thegfx942/gfx950 split-K kids would remove the per-stream registry entirely and let these rows come
back. That is the change I would rather see; this PR unblocks capture-mode users in the meantime.
Framework-side workarounds were tried first and are not sufficient — calling
opus_gemm_workspace_init()on the capture stream from vLLM(vllm-project/vllm#53238, now closed) lets the PIECEWISE ladder finish but
the abort simply relocates to FULL capture, because the registry is per-stream and the framework
cannot reliably pre-register every stream that will capture.
Test Plan
Kimi-K3 MXFP4 TP8 (bf16 GEMM path), 8x MI355X (gfx950), ROCm 7.2.3, aiter v0.1.20, vLLM main
with cudagraphs enabled,
--max-num-seqs 20(concurrency 8, 48-entry PIECEWISE ladder).Step 2 was executed by pointing
AITER_CONFIG_GEMM_BF16at a table generated by the same rule.On gfx950 that table is equivalent to this diff by construction: the lookup is arch-keyed, so the
gfx942/gfx1250 rows are not selectable on this hardware.
Test Result
opus_gemm.cu:487abortsWorker proc died unexpectedly→Engine core initialization failedConcurrency dependence: at concurrency 1 the capture ladder is small enough that no shape
routes to opus, and the stock tables pass. The failure only appears once the ladder is wide enough
to reach the affected shapes (observed at
M:20/19/18, N:6288). Low-concurrency smoke tests do notcatch this.